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AUGMENTED  REALITY  MENTOR  FOR  TRAINING  MAINTENANCE 
PROCEDURES:  INTERIM  ASSESSMENT 

Background 

The  U.S  Army  seeks  to  improve  instruetion  to  maintain  military  readiness.  One  speeifie 
instructional  challenge  focuses  on  the  training  of  maintenance  personnel  for  armored  vehicles. 
Armored  vehicle  technology  is  complex.  There  is  not  enough  time  in  the  schoolhouse  to  teach 
every  maintenance  procedure  for  every  piece  of  equipment.  Further,  instructors  are  assigned  to 
their  roles  in  temporary  rotations,  typically  lasting  up  to  3  years,  and  most  come  to  their 
instructional  task  with  limited  teaching  experience.  Given  these  constraints,  the  U.S  Army  seeks 
technologies  to  improve  the  efficiency  and  effectiveness  of  instruction  and  to  support  on-the-job 
learning. 

Recent  research  suggests  that  augmented  reality  and  automated  voice  technologies  offer 
the  potential  to  achieve  improved  learning.  Together,  these  technologies  have  the  potential  to: 

•  Improve  learning  and  long-term  memory  through  parallel  processing  of  multiple 
representations  of  information  (e.g.,  video  and  audio)  (Mayer  &  Moreno,  2002). 

•  Reduce  the  demands  on  learners’  short-term  memory  during  learning  by  presenting 
dynamic  representations  of  multi-step  procedures  (Chandler  2009;  Hegarty  et  ah,  2003; 
Lowe  1999). 

•  Improve  learners’  long-term  memory  of  multi-step  procedures  by  focusing  their  attention 
on  the  boundaries  of  key  sub-steps  (Zacks  et  ah,  2007). 

•  Build  learners’  understanding  of  key  sub-steps  by  giving  them  more  control  over  the 
review  of  these  steps  (Spanjer,  Wouters,  Van  Gog,  &  Van  Merrienboer,  2011). 

•  Focus  learner  attention  efficiently  on  the  central  elements  of  a  complex  procedure  (Yantis 
&  Jonides,  1984). 

Additionally,  such  technologies  have  the  potential  to  increase  the  time  in  the  schoolhouse 
for  higher-order  learning.  For  example,  they  may  reduce  the  burden  on  instructors  for  low- 
level,  redundant  teaching  activities  such  as  correcting  errors  in  technical  manuals  and  re¬ 
explaining  basic  procedures,  thus  freeing  up  the  opportunity  for  instruction  around  higher-order 
problem  solving.  They  may  also  reduce  the  burden  on  peer  “helpers”  who  read  technical 
manuals  aloud,  thus  increasing  the  opportunity  for  learners  to  engage  in  more  complex 
activities,  such  as  collaborative  problem  solving. 

Based  on  these  principles,  the  Army  Research  Institute  has  contracted  SRI  International 
to  develop  and  test  a  prototype  technology,  the  Augmented  Reality  Mentor  (AR  Mentor).  It  is  a 
head-mounted  display  device  through  which  technicians  can  receive  audio-visual  guidance  while 
learning  to  conduct  adjustments  on  military  equipment.  The  AR  Mentor  provides  guidance  via 
on-demand  voice  instruction  and  five  types  of  visual  overlays  to  the  work  environment:  3D 
graphic  animations  to  describe  tools  and  how  to  manipulate  them,  animated  arrows  that  direct  the 
learners’  gaze  direction,  live-action  videos  of  maintainers  conducting  adjustment  procedures, 
text-annotated  graphic  images  of  complex  tools,  and  diagrammatic  images  of  complex 
equipment. 
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The  learner  may  direet  the  flow  of  these  audio-visual  forms  of  guidanee,  skipping  them 
and  asking  them  to  repeat  as  needed.  The  deviee  additionally  features  a  limited  eapaeity  to 
deteet  when  the  learner  is  positioned  correctly  on  military  equipment  to  conduct  an  adjustment. 

The  AR  Mentor  first  prototype  was  developed  from  September  2012  through  October 
2013  to  support  instruction  around  an  advanced  adjustment  to  the  Bradley  Fighting  Vehicle  (the 
“Bradley”).  In  November  2013,  SRFs  Education  Division  designed  and  conducted  in  progress 
performance  and  usability  assessments  of  AR  Mentor.  This  report  summarizes  the  assessment 
approach  and  emerging  findings. 

The  AR  Mentor  performance  assessment  focused  on  a  set  of  proof-of-concept  research 
questions  focused  on  the  technology’s  usability  and  efficacy  for  task  learning. 

1 .  What  are  novice  Soldier  perceptions  of  the  usability  of  the  technology? 

2.  What  quality  of  task  performance  do  novice  Soldiers  demonstrate  using  the  technology  (as 
measured  by  number  of  errors,  frequency  of  help-seeking,  time  to  complete  sub-steps,  and 
time  to  complete  the  entire  procedure)? 

3.  What  are  novice  Soldiers’  perceptions  of  their  own  learning  with  the  technology? 

4.  What  are  the  perceptions  of  experienced  mechanics  of  the  usability  and  efficacy  of  the 
technology  for  learning  adjustments  on  unfamiliar  military  equipment? 

5.  What  are  the  perceptions  of  instructors  of  the  usability  and  efficacy  of  the  technology  for 
teaching  how  to  make  adjustments? 


Method 

To  obtain  an  objective  assessment  of  the  novice  Soldiers’  perceptions  of  and  performance 
with  the  AR  Mentor  prototype,  the  study  compared  pairs  of  novices  learning  the  procedure  in 
three  different  learning  conditions: 


•  AR  Mentor  only, 

•  Instructor/technical  manual,  and 

•  Technical  manual  only. 


The  instructor/technical  manual  condition  reflected  business-as-usual  instruction  in  the 
schoolhouse  as  the  instructor  monitors  a  student  pair  using  the  technical  manual  to  perform  the 
procedure.  The  technical  manual  only  condition  provided  a  baseline  view  of  learner  capacity 
unaided  by  instructor  guidance.  Pilot  testing  and  observations  of  standard  schoolhouse 
instruction  had  indicated  that  instructors  use  intrusive  guidance  with  Soldiers,  meaning  they 
monitor  the  novices’  performance  closely  and  intervene  before  novices  request  assistance.  This 
instructional  approach  can  mask  the  actual  points  of  difficulty  the  novices  encounter  in 
procedural  learning,  possibly  giving  them  a  false  sense  of  competence  and  a  limited  opportunity 
to  develop  resilience  in  problem  solving  (Shechtman  et  ah,  2013). 
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Additionally,  we  engaged  both  Bradley  instruetors  and  trained  tank  meehanies  with  no 
Bradley  experienee  in  using  the  AR  Mentor  to  eonduet  the  repair  proeedure,  and  then 
interviewed  them  about  their  perceptions  of  its  usability  and  efficacy. 

Participants 

We  engaged  three  types  of  participants:  12  novice  Bradley  maintainers,  three 
experienced  tank  maintainers  who  had  little  to  no  experience  with  Bradleys,  and  three 
experienced  Bradley  instructors.  All  were  recruited  in  accordance  with  human  subjects 
regulations  and  procedures. 

The  novice  Bradley  maintainers  were  selected  for  their  lack  of  exposure  to  the  procedure. 
For  all  but  one  pair,  none  of  the  novices  had  ever  been  taught  this  specific  procedure  and  none 
had  actually  performed  it  before.  They  were  also  selected  to  provide  a  representative  range  of 
learner  aptitude  based  on  their  scores  on  subtests  of  the  Armed  Services  Vocational  Aptitude 
Battery  (ASVAB).  For  Bradley  maintainers,  the  composite  tests  were  the  Mechanical 
Maintenance  (MM),  which  combines  the  ASVAB  subtests  of  numerical  operations,  auto  and 
shop  information,  mechanical  comprehension,  and  electronics  information,  and  the  General 
Technical  (GT),  which  combines  the  ASVAB  subtests  of  Paragraph  Comprehension  and 
Arithmetic  Reasoning.  As  a  reference,  the  Army  recommends  that  maintainers  of  Bradley 
Fighting  Vehicles  have  a  minimum  MM  score  of  105.  Our  sample  included  a  range  of  novices 
who  scored  a  low  of  89  and  a  high  of  134  on  the  MM.  It  also  included  one  Soldier  for  whom 
English  was  a  second  language  and  one  Soldier  for  whom  English  was  not  the  primary  language 
spoken  at  home.  Each  condition  included  a  pair  with  higher  ASVAB  scores  and  a  pair  with 
lower  ASVAB  scores. 

Procedures 

We  engaged  all  participants  in  performing  a  34-step  adjustment  procedure.  Pairs  of 
novice  Soldiers  and  experienced  maintainers  conducted  the  procedure  collaboratively  in  each 
learning  condition.  Instructors  were  on  hand  in  all  three  conditions  to  intervene  if  students  made 
an  error  that  would  impede  progress,  but  they  were  asked  to  avoid  intervening  in  the  AR  Mentor 
and  manual  conditions,  except  for  providing  guidance  as  they  normally  would  in  the  instructor 
condition. 

The  instructors  conducted  the  procedure  independently  with  physical  help  from  the 
research  team.  In  the  AR  Mentor  condition  only,  both  members  of  a  pair  had  a  chance  to  use  the 
AR  Mentor  so  the  research  team  could  collect  more  data  on  usability  and  efficacy.  In  the  other 
two  learning  conditions,  the  pairs  went  through  the  procedure  only  once  to  provide  comparative 
performance  data.  We  allocated  up  to  2-3  hours  for  the  performance  task  and  post-procedure 
interviews.  All  repairs  were  conducted  on  Bradleys  in  a  military  schoolhouse  and  set  to  the  same 
starting  conditions. 

Instrumentation 

We  developed  two  protocols  for  all  novice  Soldiers,  an  observational  protocol  (Appendix 
A)  and  a  learning  experience  questionnaire  (Appendix  B).  In  addition,  for  the  AR  Mentor 
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condition,  we  developed  the  AR  Mentor  usability  questionnaire  (Appendix  C)  and  an  interview 
protocol  that  was  administered  to  the  Soldiers  in  that  eondition  only  (Appendix  D).  The 
observation  protoeol  reeorded  the  following  performance  data:  time  to  complete  three  sub-phases 
of  the  adjustment,  total  task  completion  time,  number  of  errors,  and  number  of  instanees  of  help 
(either  sought  or  intrusively  provided).  In  the  instructor  condition  only,  we  also  tallied  the  types 
of  instructor  guidance  provided  (procedural,  conceptual,  self-regulating,  safety  precaution, 
technical  manual  correction).  The  learning  experienee  questionnaire  included  a  7-item  holistic 
rating  of  perceived  diffieulty  of  the  learning  experienee  using  a  modified,  unweighted  version  of 
the  NASA  Task  Load  Index  (Hart  &  Staveland  1988)  and  a  researeher-designed  17-item  rating 
of  pereeived  diffieulty  of  each  of  the  steps  of  the  proeedure  (Nederhof  1985).  This  instrument 
employed  a  third-person  diffieulty  rating  strategy  to  minimize  soeial  stigma  (e.g.,  “You  need  to 
tell  another  Soldier  whieh  steps  are  harder. . .),  and  then  presented  pairs  of  steps  from  the 
proeedure.  The  steps  varied  on  procedural  or  eoneeptual  eomplexity.  These  pairs  were  derived 
from  absolute  ratings  of  each  step  by  both  researchers  (based  on  observations  of  Soldiers  doing 
the  task)  and  an  experieneed  Bradley  instructor.  The  AR  Mentor  usability  questionnaire 
included  19  Likert-seale  items  asking  for  ratings  of  the  ease  of  using  the  teehnology’s  visual  and 
audio  guidance  features.  The  AR  Mentor  usability  interview  protoeol  ineluded  three  questions 
seeking  more  detail  about  each  participant’s  preferences  for  audio  and  visual  guidanee  features 
and  experience  of  physical  comfort  using  the  technology. 


Results 

This  report  focuses  primarily  on  the  quantitative  results  from  the  data  collection,  and  a 
future  report  will  focus  more  on  qualitative  analysis  and  a  closer  look  at  the  step-by-step  learning 
efficaey  results.  For  this  report,  there  were  two  primary  analyses  we  condueted  on  the  data  from 
the  novice  Soldiers:  Comparison  of  their  performanee  metries  and  comparison  of  their  holistie 
pereeptions  of  learning  efficacy  among  the  three  conditions.  We  additionally  analyzed  the 
holistie  learning  efficaey  results  according  to  Soldiers’  ASVAB  seores  (MM  and  GT).  This 
analysis  clarified  how  Soldiers  of  different  verbal,  quantitative,  and  meehanieal  aptitude  varied 
in  their  perceptions  of  the  diffieulty  of  learning  the  task  in  the  three  different  eonditions.  For  this 
report,  we  also  analyzed  feedback  from  the  AR  Mentor  novices  for  their  ratings  of  the  overall 
usability  of  the  teehnology  and  the  speeific  usability  of  the  teehnology’s  visual  and  the  audio 
guidanee  features.  We  also  eompleted  brief  initial  qualitative  summaries  of  feedback  from  the 
experienced  mechanics  and  the  instructors  about  the  usability  of  the  AR  Mentor.  The  samples  in 
this  study  are  small,  and  so  the  results  should  be  considered  exploratory. 

We  present  the  noviee  Soldier  results  first.  This  includes  the  performance  metrics 
comparison,  followed  by  the  comparison  of  holistic  perceptions  of  learning  efficacy,  and 
concluding  with  the  AR  Mentor  usability  data. 

Performance  Metrics  Comparison 

Counts  of  performance  metrics  showed  that  the  AR  Mentor  eondition  took  slightly  longer 
and  involved  a  few  more  errors  than  the  instructor  condition.  Novice  Soldiers  took  longer  and 
made  more  errors  in  the  manual-only  condition.  The  data  on  help  seeking  and  total  instances  of 
instruetor  intervention  indicate  that  the  AR  Mentor  required  much  less  guidanee  from  the 
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instructor  (see  Table  1).  The  noviees  working  with  the  AR  Mentor  were  observed  to  be  engaged 
in  mueh  more  independent,  eollaborative  troubleshooting  than  those  in  the  instruetor  condition. 
The  trends  from  the  observational  data  will  be  presented  in  a  future  report. 

Table  1. 

Comparison  of  average  total  trainee  error,  help-seeking,  instructor  guidance,  and  total  time 
for  the  three  learning  conditions* 


Learning 

Condition 

Total  Errors 
Mean 

(individual  scores) 

Total  Help 
Seeking 

Mean 

(individual  scores) 

Total  Instructor 
Guidance 

Mean 

(individual  scores) 

Average  Total 
Time 

(hour:min:see) 

AR  Mentor 

2.75 

7.50 

2.00 

1:11:00 

(n=4) 

(5, 2,3,1) 

(15,3,11,1) 

(6,1, 1,0) 

Instructor  + 

Manual 

(n=2) 

1.50 

(1,2) 

8.00 

(9,7) 

46.50 

(58,35) 

0:55:30 

Manual  only 

8.00 

25.00 

22.50 

2:15:00 

(n=2) 

(11,5) 

(25,25) 

(26,19) 

*  To  see  the  full  distributions  see  appendix  E 


Holistic  Perceptions  of  Learning  Efficacy 

The  NASA  Task  Load  Index  results  are  presented,  foeusing  on  the  Soldiers’  perceptions 
of  the  mental  and  physical  demand  of  learning  the  proeedure,  the  pace  of  the  learning  and  related 
frustration  levels.  Then,  we  present  an  aggregate  rating  of  the  Soldiers’  perceptions  of  overall 
difficulty  of  learning  in  each  condition.  These  analyses  also  explore  how  students’  pereeptions 
of  learning  vary  by  their  baseline  competence,  as  indicated  by  their  ASVAB  seores. 

Perceptions  of  Mental  and  Physical  Demand  by  Condition 

The  novice  Soldiers  reported  lower  levels  of  both  mental  and  physical  demand  in  the  AR 
Mentor  eondition  compared  to  the  instructor  condition  and  the  manual  condition.  The  table 
below  presents  the  average  rating  the  students  reported  on  the  following  two  questions  on  the 
survey:  How  mentally  demanding  was  learning  the  adjustment?  How  physically  demanding  was 
learning  the  adjustment?  (1-5,  Very  low  to  Very  high). 

Table  2. 

Comparison  of  novice  Soldiers’  mean  perceptions  of  the  mental  and  physical  demand  while 
learning  a  technical  procedure  in  the  three  conditions 


AR  Mentor 
n  =4 

Manual  and 
Instructor 
n  =4 

Manual  Only 
n  =4 

Mental  demand 

1.25 

1.75 

3.25 

Physical  demand 

1.75 

3.00 

2.75 

5 


To  gain  insight  into  these  scores,  it  helps  to  look  at  the  distribution  of  results  (Figures  1 
and  2).  The  distribution  of  the  ratings  in  the  Manual  Only  condition  is  the  widest  for  both  forms 
of  demand,  mental  and  physical.  Only  one  student  reported  low  mental  demand  (1)  while  all 
others  reported  higher  scores  (3  or  greater).  The  distribution  of  the  physical  demand  ratings  is 
higher  in  the  Manual  Only  and  Instructor  conditions  than  in  the  AR  Mentor  condition. 


Figure  1.  Distributions  of  novice  Soldiers’  perceptions  of  mental  demand  in  three  different 
learning  conditions;  AR  Mentor,  Instructor,  and  Technical  Manual. 
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AR  Mentor  Manual  and  Instructor  Manual  only 

Learning_Conditions 


*  Each  dot  represents  a  student  in  each  condition. 

Figure  2.  Distributions  of  novice  Soldiers’  perceptions  of  physieal  demand  in  the  three  different 
learning  conditions. 

Perceptions  of  Pace  of  Learning  and  Frustration 

The  novice  Soldiers  reported  highest  levels  of  frustration  when  learning  in  the  manual 
only  condition  and  lowest  in  the  instructor  condition,  while  they  rated  the  AR  Mentor  between 
these  two  extremes  (see  Table  3).  They  reported  the  pace  was  slowest  in  the  manual  condition, 
fastest  in  the  instructor  condition,  and  they  rated  the  AR  Mentor  between  these  points. 


Table  3. 

Comparison  of  novice  Soldiers  ’  mean  perceptions  of  the  pace  of  learning  and  frustration 
levels  while  learning  a  technical  procedure  in  the  three  different  conditions _ 


AR  Mentor 

Manual  and 
Instructor 

Manual  Only 

Pace  of  Learning 
(l=slow,  5=fast) 

2.75 

3.25 

2.00 

Frustration 
(l=low,  5=high) 

2.25 

1.00 

4.00 

We  hypothesized  that  the  rating  for  the  pace  of  learning  was  tied  to  some  of  the  other 
underlying  variables,  such  as  their  frustration  level.  Exploring  their  ratings  of  the  pace  of 
learning  with  the  overlay  of  the  frustration  variable  shows  that  students  who  have  experienced 


7 


higher  level  of  frustration  tended  to  pereeive  the  paee  of  learning  as  slow,  while  those  students 
(with  the  exeeption  of  one)  who  reported  lower  levels  of  frustration  tended  to  report  faster  paee 
of  learning  experienee  (see  Figure  3). 


TT7TTT7T7777T7T7T77T7777TT7T77T77TT7TrJTr777T77T7T777TT7TTTT7T7TT7TTTT77TTT7TrrrrT7TT77T7T7Trrr77TTT777TT7T 


Learning  Experience 

■  Dot  Plot  1  ;  j 

6 

5 

■ 

Qi 

1'* 

■ 

« 

0) 

A  3- 

■ 

■  ■■ 

82. 

■ 

■  ■■■ 

Q. 

1 

■ 

AR  Mentor 

Manual  and  Instructor  Manual  only 

Frustration 

Learning_Conditions 

0  1 

2  3  4  5  6 

*  Each  dot  represents  a  student  in  each  condition. 


Figure  3.  Comparison  of  how  noviee  Soldiers’  frustration  levels  relate  to  their  pereeptions  of 
the  paee  of  learning  by  the  three  different  learning  eonditions. 


Overall  Perception  of  the  Challenge  of  Learning  the  Task 

All  the  faeets  of  the  NASA  Task  Load  Index  (exeept  for  pereeptions  of  sueeess)  ean  be 
linked  to  an  overall  measure  of  pereeived  ehallenge  of  the  task  (1-5,  Very  low  to  Very  high). 
This  combines  perceptions  of  mental  demand,  physical  demand,  effort,  frustration,  difficulty  of 
asking  questions  (reversed  scale),  and  pace  of  learning  (reversed  scale).  The  distribution  shows 
that  novice  Soldiers  perceived  learning  with  the  AR  Mentor  to  be  comparably  challenging  to 
learning  with  an  instructor.  Novice  Soldiers  reported  both  of  these  conditions  were  easier  than 
learning  with  the  manual  only  (see  Table  4). 

Table  4. 

Comparison  of  novice  Soldiers’  mean  perceptions  of  the  overall  challenge  of  learning  a 
technical  procedure  in  the  three  different  conditions 


AR  Mentor 

Manual  and 
Instructor 

Manual  Only 

Average  Challenge 

2.29 

2.04 

3.46 

We  again  provide  the  distribution  of  results  to  clarify  the  nature  of  these  ratings.  There  is 
a  wider  distribution  in  the  manual  condition  and  the  AR  Mentor  condition  compared  to  the 
instructor  condition  (see  Figure  4). 


AR  Mentor 


Manual  and  Instructor  Manual  only 

Learning_Conditions 


Each  dot  represents  a  student  in  each  condition. 


Figure  4.  Distributions  of  novice  Soldiers’  perceptions  of  the  challenge  of  learning  a  procedural 
task  in  three  different  learning  conditions;  AR  Mentor,  Instructor,  and  Technical  Manual. 

Exploration  of  Holistic  Ratings  of  Learning  Efficacy  with  Novice  Soldiers’  ASVAB  Scores 

This  analysis  helped  explain  some  of  the  wider  distributions  in  novice  Soldiers’ 
perceptions  of  the  difficulty  of  learning  in  the  different  learning  conditions.  As  Figure  5  shows, 
students  with  a  lower  ASVAB  scores  tended  to  perceive  the  task  as  more  mentally  demanding 
than  those  with  higher  ASVAB  scores  in  all  three  conditions,  but  particularly  in  the  manual  only 
condition  (see  Figure  5). 
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Figure  5.  Distributions  of  novice  Soldiers’  perceptions  of  the  mental  demand  of  learning  a 
procedural  task  in  the  three  different  learning  conditions  according  to  Soldiers’  ASVAB  scores. 


By  eontrast,  novice  Soldiers’  pereeptions  of  the  physical  demand  of  the  task  did  not 
appear  to  relate  as  strongly  to  their  ASVAB  scores  (see  Figure  6). 
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Each  dot  represents  a  student  in  each  condition. 


Figure  6.  Distributions  of  novice  Soldiers’  perceptions  of  the  physical  demand  of  learning  a 
procedural  task  in  the  three  different  learning  conditions  according  to  Soldiers’  ASVAB  scores. 
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Novice  Soldiers’  pereeptions  of  the  level  of  effort  they  devoted  to  eompleting  the  task 
also  did  not  appear  to  relate  strongly  to  their  ASVAB  seores  (see  Figure  7). 


Figure  7.  Distributions  of  noviee  Soldiers’  pereeptions  of  level  of  effort  of  learning  a  proeedural 
task  in  the  three  different  learning  conditions  according  to  Soldiers’  ASVAB  scores. 

Noviee  Soldiers’  pereeptions  of  the  diffieulty  of  asking  questions  while  learning  the  task 
varied  by  both  the  learning  eondition  and  their  ASVAB  seores.  Students  with  low  ASVAB 
scores  reported  higher  diffieulty  asking  questions  of  the  AR  Mentor  than  those  with  higher 
ASVAB  scores.  Sueh  differenees  were  not  observed  in  the  Manual  and  Instructor  condition. 
There  was  a  wider  range  of  pereeptions  of  diffieulty  asking  questions  reported  in  the  manual 
eondition  (see  Figure  8). 
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Figure  8.  Distributions  of  novice  Soldiers’  perceptions  of  the  difficulty  of  asking  questions 
while  learning  a  procedural  task  in  three  different  learning  conditions  AR  Mentor,  Instructor,  and 
Technical  Manual  according  to  Soldiers’  ASVAB  scores. 

Novice  Soldiers’  reports  of  their  frustration  levels  with  the  task  also  varied  by  both  the 
learning  condition  and  their  ASVAB  scores.  Students  with  both  high  and  low  ASVAB  scores 
reported  relatively  low  levels  of  frustration  in  the  instructor  condition,  but  relatively  high  levels 
of  frustration  in  the  manual  condition,  with  the  ASVAB  in  the  middle  (see  Table  5).  The 
distribution  of  frustration  level  is  widest  in  the  AR  Mentor  condition,  and  low  scorers  are  split 
between  highly  frustrated  and  minimally  frustrated,  while  higher  ASVAB  scorers  are  both 
reporting  relatively  low  levels  of  frustration  (see  Figure  9). 

Table  5. 

Comparison  of  novice  Soldiers  ’  mean  perceptions  of  frustration  while  learning  a  technical 
procedure  in  three  different  conditions:  AR  Mentor,  Instructor,  and  Technical  Manual 
according  to  Soldiers’  ASVAB  scores 


AR  Mentor 

Manual  and 
Instructor 

Manual  Only 

Frustration 

2.25 

1. 00 

4.00 

*  Each  dot  represents  a  student  in  each  condition. 

Figure  9.  Distributions  of  novice  Soldiers’  perceptions  of  frustration  while  learning  a  procedural 
task  in  three  different  learning  conditions  AR  Mentor,  Instructor,  and  Technical  Manual 
according  to  Soldiers’  ASVAB  scores. 
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AR  Usability  Data 

We  present  analyses  of  the  overall  ratings  of  AR  Mentor’s  usability,  followed  by  ratings 
of  the  usability  of  the  visual  representations  and  the  audio  guidance,  and  the  physical  comfort 
aspects  of  using  the  technology. 

Overall  usability.  Novice  Soldiers  gave  high  ratings  to  the  overall  usability  for  AR 
Mentor.  They  reported  feeling  they  and  their  helpers  knew  what  adjustments  they  needed  to 
make  and  that  the  AR  Mentor  provided  a  smooth  flow  of  guidance  (see  Table  6). 

Table  6. 

Novice  Soldiers^  perceptions  of  overall  usability  of  the  AR  Mentor 


Overall  Usability 

Soldier  IE 

) 

Pl.l 

P1.2 

P2.2 

P2.3 

Scale  of  1-5  (Low  to  High) 

Average 

Rating 

How  well  could  you  understand  what  you 
needed  to  do  to  adjust  the  Bradley? 

5 

2 

4 

5 

4.00 

How  consistently  did  the  AR  Mentor 
audio  and  visual  instructions  occur  when 
you  needed  them  to  occur? 

4 

4 

5 

4 

4.25 

How  consistently  did  you  know  which 
steps  required  you  to  coordinate  with  a 
helper  and  which  required  action  only  by 
you? 

4 

3 

5 

5 

4.25 

Overall 

4.17 

Visual  representations.  The  AR  Mentor  provides  visual  guidance  using  five  types  of 
visual  representations  that  are  overlaid  on  the  work  environment:  3D  graphic  animations  to 
describe  tools  and  how  to  manipulate  them,  animated  arrows  that  direct  the  learners’  gaze 
direction,  live-action  videos  of  maintainers  conducting  adjustment  procedures,  text-annotated 
graphic  images  of  complex  tools,  and  diagrammatic  images  of  complex  equipment  and  the 
Bradley  location  map. 

The  novice  Soldiers  (n=4)  gave  generally  high  ratings  to  the  understandability  of  the  AR 
Mentor’s  visual  representations.  Soldiers  showed  a  slight  preference  for  videos  of  a  maintainer 
conducting  a  repair.  “It’d  be  almost  like  having  another  instructor  there  since  it’s  showing  them 
exactly  what  they  need  to  do,  the  tools  they  need  to  have  in  place,  exactly  what  they  need,”  one 
instructor  said.  Ratings  of  the  understandability  of  these  different  representations  are  presented 
in  Table  7. 
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Table  7. 

Novice  Soldiers^ perceptions  of  understandability  of  AR  Mentor’s  visual  representations 


Visual  Representations 

Soldier  ID 

Pl.l 

P1.2 

P2.2 

P2.3 

Scale  of  1-5  (Low  to  High) 

Average 

Rating 

Video  demos 

5 

4 

5 

5 

4.75 

Text 

4 

4 

5 

5 

4.5 

Direetional  arrows 

NA 

4 

4 

5 

4.33 

Diagrams 

4 

3 

5 

5 

4.25 

Bradley  map  image 

4 

3 

5 

5 

4.25 

3D  animations 

5 

3 

5 

4 

4.25 

Overall 

4.39 

The  direetional  arrows  and  Bradley  map  image  provide  Soldiers  with  visual  pointers  to 
where  they  should  be  loeated  when  performing  sub-steps  of  the  adjustment.  Soldiers  generally 
reported  being  able  to  find  where  they  should  be  standing  on  the  Bradley  when  using  the  AR 
Mentor  (Table  8). 

Table  8. 

Novice  Soldiers’  perceptions  of  ease  of  determining  where  to  stand  on  the  Bradley  while  using 
the  AR  Mentor 


Orientation 

Soldier  ID 

Pl.l 

P1.2 

P2.2 

P2.3 

Seale  of  1-5  ( 

'Low  to : 

High) 

Average 

Rating 

How  easy  was  it  to  find  where  you  should 
be  standing  on  the  Bradley? 

5 

2 

5 

5 

4.25 
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Automatic  Speech  Recognition  (ASR)  &  Virtual  Personal  Assistant  (VP A), 

The  AR  Mentor  features  a  realistic  male  voice  that  guides  the  learner  through  each  task.  The 
voice  is  projected  so  that  the  “helper”  and  instructor  can  hear  what  the  Soldier  wearing  the  head- 
mounted  rig  can  hear.  The  system  is  programmed  to  permit  the  Soldiers  to  engage  in  regular 
conversation  and  to  respond  only  to  direct  queries  prefaced  by  the  term,  “Computer,”  as  in, 
“Computer,  please  go  to  step  10”.  For  each  step,  the  VPA  goes  through  a  script  describing  the 
procedures.  These  are  chunked  at  different  levels  of  information  complexity,  ranging  from  a 
single  step  to  roughly  five  sub-steps.  Safety  warnings  are  inserted.  The  VPA  additionally 
checks  with  the  Soldier  for  understanding  and  completion  of  each  step  before  proceeding  to  the 
next  step.  It  is  programmed  to  respond  to  a  range  of  queries,  from  simple  procedures  to  open- 
ended  queries  focused  on  the  purpose  of  the  task.  When  the  VPA  system  fails  to  comprehend  a 
request  or  question,  it  says  so  and  asks  the  Soldier  to  restate. 

The  ASRWPA  system  has  distinct  interactive  dimensions:  Speech  recognition,  question 
response,  and  pace. 

The  novice  Soldiers  gave  moderately  high  ratings  to  the  system  for  speech  recognition 
(see  Table  9). 

Table  9. 

Novice  Soldiers’  perceptions  of  quality  of  AR  Mentor’s  speech  recognition 


Speech  Recognition 

Soldier  ID 

Pl.l 

P1.2 

P2.2 

P2.3 

Scale  of  1-5  (Low  to 
High) 

Average 

Rating 

How  often  did  the  AR  Mentor  know  when  you  were 
speaking? 

4 

4 

3 

3 

3.50 

How  well  did  you  understand  the  AR  Mentor’s  audio 
instmctions? 

4 

3 

4 

5 

4.00 

How  often  did  the  AR  Mentor  respond  correctly  to 
what  you  were  saying? 

5 

3 

4 

4 

4.00 

Overall 

3.83 

Too  few  Soldiers  asked  questions  of  the  system  to  get  a  sense  of  its  quality  (see  Table  10). 
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Table  10. 

Novice  Soldiers’  perceptions  of  quality  of  AR  Mentor’s  question  response 


Question  Response 

Soldier  ID 

PI. I 

P1.2 

P2.2 

P2.3 

Scale  of  1-5  (Low  to 
High) 

Average 

Rating 

If  you  asked  the  AR  Mentor  any  questions,  how 
relevant  and  useful  were  the  AR  Mentor’s 
answers? 

1 

n/a 

n/a 

n/a 

1.00 

If  you  asked  AR  Mentor  to  go  to  a  step,  how 
well  could  the  AR  Mentor  go  to  the  step  you 
wanted? 

5 

n/a 

5 

n/a 

5.00 

Novice  Soldiers  generally  expressed  impatience  with  the  pace  of  the  voice  guidance 
(see  Table  11). 

Table  11. 

Novice  Soldiers’  perceptions  of  quality  of  AR  Mentor’s  pace  of  voice  guidance 


Pace  of  AR  Mentor  voice  guidance 

Soldier  ID 

PI. I 

PI. 

2 

P2.2 

P2.3 

Average 

Rating 

Scale  of  1-5  (Always  to 
Never) 

How  often  did  you  want  to  interrupt  the  AR 
Mentor  when  it  was  saying  it  didn’t 
understand  you? 

2 

3 

1 

2 

2.00 

How  often  did  you  want  to  speed  up  the  AR 
Mentor  voice  when  it  was  speaking? 

2 

1 

2 

1 

1.50 

How  often  did  you  want  to  slow  down  the  AR 
Mentor  voice  when  it  was  speaking? 

4 

5 

3 

5 

4.25 

Overall 

2.58 

Physical  Aspects:  Comfort,  In  terms  of  comfort,  Soldiers  gave  mid-range  ratings  to  the 
extent  to  which  the  AR  Mentor  restricted  their  movement  and  got  in  the  way  of  their  body  and 
head  (see  Table  12). 
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Table  12. 

Novice  Soldiers’  perceptions  of  comfort  when  using  the  AR  Mentor 


Physieal  movement  using  AR  Mentor 

Soldier  ID 

Pl.l 

PI.2 

P2.2 

P2.3 

Seale  of  1-5  (Never  to 
Always) 

Averag 

e 

Rating 

To  what  extent  did  the  AR  Mentor  restriet  your 
movement  and  get  in  the  way  of  your  body  and 

3 

2 

3 

4 

3.00 

your  head? 

Summary  of  Instructor  and  Experienced  Mechanic  Perspectives 

In  interviews  eondueted  after  using  the  AR  Mentor  to  eonduet  the  proeedure,  both 
Bradley  instruetors  and  experieneed  tank  meehanies  reported  generally  positive  experienees  with 
the  teehnology.  This  seetion  briefly  summarizes  those  responses. 

All  three  Bradley  instruetors  endorsed  the  use  of  visual  representations  to  guide  the 
learning  of  the  proeedure  and  eharaeterized  the  teehnology  as  a  type  of  “baekup  instruetor”  who 
eould  remind  students  what  to  do.  They  liked  the  VPA’s  eapaeity  to  require  students  to  hear  the 
safety  preeautions.  They  all  said  their  students  would  like  using  it.  They  eould  see  the  tool  as  a 
way  to  engage  more  learners  in  learning,  either  by  freeing  up  the  instruetor’s  time  to  foeus  on 
higher-level  aspeets  of  troubleshooting  or  adding  an  additional  eapability  to  the  teehnology  that 
would  permit  other  students  to  see  on  a  large  sereen  what  the  hands-on  student  was  seeing  while 
eondueting  an  adjustment.  Two  noted  the  AR  Mentor  restrieted  mobility  and  visibility,  but  not 
to  the  point  of  preventing  the  work  to  proeeed.  They  said  they  would  like  to  see  the  teehnology 
eover  all  the  eontent  in  the  teehnieal  manual. 

All  three  experieneed  tank  meehanies  liked  the  AR  Mentor  as  a  way  to  walk  through  a 
new  proeedure  in  the  field.  “If  I  ean  see  it,  I  ean  do  it,”  as  one  said,  and  the  teehnology 
eapitalized  on  this  preferenee.  They  said  they  preferred  learning  new  proeedures  from  sueh 
teehnology  to  a  teehnieal  manual,  and  expressed  the  belief  that  they  eould  better  reeall  a 
proeedure  after  just  one  pass  with  the  AR  Mentor.  They  expressed  reservations  about  the 
teehnology  for  eondueting  repairs  in  tight  spaees  or  bright  light.  They  reeommended  a  tighter, 
more  eompaet  form.  They  also  said  the  AR  Mentor  voiee  was  too  slow,  and  they  would  prefer  to 
eontrol  it  with  a  push  button.  In  partieular,  they  did  not  want  to  wait  to  go  through  every  new 
tool  during  the  initial  tool  eheek;  rather,  they  would  like  to  see  all  the  tools  and  then  press  an 
“OK”  button.  All  said  the  AR  Mentor  built  on  their  own  knowledge  of  tools  and  moving  around 
large  military  armored  vehieles  effeetively.  Although  the  experieneed  meehanies  had  either 
never  worked  on  a  Bradley  before  or  eondueted  an  unrelated  adjustment  onee  in  the  distant  past, 
eaeh  eould  eonduet  the  repair  in  under  an  hour  on  the  first  pass  with  the  AR  Mentor. 
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Conclusions 


Overall,  all  the  partieipants — noviee  Soldiers,  Bradley  instruetors,  and  experieneed 
meehanies — gave  generally  positive  ratings  to  the  AR  Mentor  prototype  and  eoneept. 
Partieularly,  they  like  the  graphies  and  video  beeause,  eonsistent  with  learning  theory,  these 
representations  required  less  mental  effort  than  the  use  of  teehnieal  manuals  to  learn  new 
proeedures.  The  data  suggest  that  the  teehnieal  manuals  present  a  partieular  ehallenge  to 
Soldiers  entering  at  the  lower  ASVAB  levels,  and  those  Soldiers  pereeive  the  AR  Mentor  to 
present  information  in  a  less  demanding  manner.  Instruetors  eould  immediately  see  that  the  time 
they  must  devote  to  redundant  aspeets  of  teaehing  eould  be  greatly  redueed  by  sueh  teehnology, 
perhaps  freeing  them  up  to  engage  in  higher-level  forms  of  instruetion. 

At  the  basie  performanee  level,  the  AR  Mentor  produeed  a  eomplete  rehearsal  of  the  full 
proeedure  in  a  slightly  longer  time  than  instruetor-led  training,  but  it  did  so  in  a  manner  that 
appeared  to  afford  a  higher  opportunity  for  the  noviees  to  engage  in  eollaborative 
troubleshooting  than  the  instruetor  eondition.  Given  the  Army’s  desire  to  improve  readiness  at 
all  levels  of  its  forees,  this  tradeoff  between  putting  slightly  more  time  into  instruetion  in 
exehange  for  a  greater  ehanee  to  develop  independent  troubleshooting  skill  may  be  worthwhile. 
Further  testing  is  needed  to  measure  how  robust  the  learning  results  are  as  eompared  to  eurrent 
sehoolhouse  methods. 

The  teehnology  generally  funetioned  well  both  for  noviee  Soldiers  with  little  meehanieal 
experienee  and  for  experieneed  meehanies  with  little  familiarity  of  the  speeifie  proeedures  to 
adjust  the  Bradley.  Our  initial  review  of  the  data  indieated  there  were  some  distinetions  in  how 
experieneed  and  inexperienee  meehanies  used  the  AR  Mentor  teehnology.  Clearly,  the  paee  of 
the  VPA  guidanee  is  something  that  would  need  to  be  adjusted  for  these  two  types  of  learners. 
For  example,  it  may  be  useful  to  require  noviees  to  hear  every  safety  preeaution  and 
aeknowledge  every  tool  during  an  initial  tool  eheek,  but  experieneed  meehanies  will  be  impatient 
with  iteration  of  sueh  obvious  basies. 

As  a  prototype,  AR  Mentor  has  limitations,  whieh  all  partieipants  aeknowledged.  A  more 
eompaet  and  robust  form  would  be  required  to  permit  meehanies  to  work  in  tight  spaees.  The 
AR  images  may  not  be  visible  through  the  monoele  in  all  environments,  partieularly  in  areas 
with  high  levels  of  light.  Some  users  found  it  ehallenging  to  adjust  to  looking  at  their  work 
through  a  monoele.  Others,  partieularly  low  ASVAB  learners,  showed  evidenee  of  diseomfort 
asking  questions  of  the  AR  Mentor.  In  faet,  not  one  Soldier  posed  a  eoneeptual  question  to  the 
AR  Mentor.  The  one  Soldier  who  tried  asking  an  open-ended  question  did  not  reeeive  a 
produetive  response.  The  AR  Mentor’s  Reasoner  system,  whieh  links  visual  representations  and 
audio  guidanee,  still  falls  short  of  its  stated  goal  to  engage  learners  in  sueh  potentially  rieh 
question-response  aetivities. 

As  a  first  step  toward  that  goal,  two  issues  are  that  some  of  the  AR  Mentor  visual 
representations  were  not  funetioning  effeetively  and  some  of  the  VPA  instruetions  eontained  too 
many  steps.  If  instruetional  eontent  is  ever  to  be  produeed  at  mass  seale  for  sueh  teehnology, 
further  study  is  needed  to  define  the  design  features  of  visual  representations  and  dialog  seripts. 
Sueh  study  needs  to  foeus  both  on  how  to  design  to  reduee  eognitive  load,  but,  more  importantly. 
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also  to  understand  what  visual  representations  and  audio  dialogs  may  serve  as  a  springboard  to 
deeper  learning. 

The  AR  Mentor  system  appears  to  reduee  the  load  on  Soldiers’  working  memory  of 
Soldiers,  as  indicated  by  the  reports  from  all  the  study  participants.  However,  this  study  raises 
questions  about  whether  such  technology  can  do  more.  In  theory,  the  AR  Mentor  technology  can 
support  improved  long-term  memory  retention  and  knowledge  retrieval.  We  still  need  to  test 
whether  using  AR  Mentor  does  improve  retention  of  procedural  knowledge.  In  addition,  a  goal 
for  the  AR  Mentor  is  to  improve  understanding  of  higher-order  concepts  that  may  improve 
troubleshooting  skill.  In  this  study,  we  did  not  observe  any  instruction  that  would  be  expected  to 
improve  conceptual  knowledge  of  mechanical  or  electrical  systems  nor  did  we  observe  any  such 
improvement.  These  are  the  questions  to  be  explored  further  through  design  and  testing. 
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Appendix  A:  Observation  Protocol 


OBSERVER: _ DATE: _ MAINTAINER  ID: _ 

MAINTAINER  GROUP: _ 

AR  MENTOR  Observation  Protocol  -  AR  MENTOR  CONDITION 

Updated  10/18/2013 

Observation/Instrument  Instructions 

•  Record  the  start  time  of  the  activity. 

• 

n  the  first  column  of  checkboxes  labeled  Complete,  indicate  with  a  check  mark  when  the  maintainer  completes  the  step. 

o 

f  the  maintainer  does  not  perform  the  step  at  any  point,  leave  the  box  empty. 

•  If  the  maintainer  backtracks  on  a  step  or  performs  a  step  incorrectly,  place  an  X  in  the  box  next  to  the  step  in  the  Error  column. 

•  Record  the  end  time  of  the  activity. 

•  Conduct  the  post-activity  conceptual  checks  (after  each  phase). 

•  Answer  the  post-observation  debrief  questions  (note:  there  are  two  sets  of  questions,  one  for  the  AR-Mentor  condition  and  one  for  the 
instructor  condition). 

Conceptual  Check: 

At  the  end  of  the  three  evaluation  phases  (setup,  continuity,  reassembly),  ask  the  maintainer  the  questions  listed  and  indicate  if  he/she  answered 
correctly  or  incorrectly  in  the  space  provided. 

Help  Seeking  Behavior. 

Place  an  “X”  in  the  column  marked  Help  Seeking  Behavior  for  any  observed  instances  of  help  seeking  behavior.  We  operationalize  help  seeking 
as: 

□  Asking  helper  for  assistance 

□  Verbally  expressing  puzzlement  or  confusion  AND  receiving  a  helper  response 

□  Showing  evidence  of  an  active  effort  to  resolve  a  problem 
It  is  not: 

□  Verbally  expressing  frustration 

□  Observing  that  one  is  puzzled  without  a  helper  response  or  active  effort  to  resolve  problem 
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Please  note  whether  problem  was  resolved. 

□  It  Is  not  asking  what  or  when  about  the  next  step. 

Instructor  Guidance. 

Note  all  Instances  of  Instructor  feedback  /  guidance,  categorizing  Instructor  guidance  In  the  following  ways: 

□  (P)  Procedural:  Related  to  how  a  task  Is  understood  or  performed 

□  (C)  Conceptual:  Provides  Information  specific  to  underlying  tasks  or  relating  and  extending  tasks  (I.e.,  encourages  deep  understanding) 

□  (SR)  Self-regulating:  Encourages  self-regulation  or  reflection  on  one’s  understanding  and  behavior 

□  (SP)  Safety  precaution:  Warns  the  student  about  safety 

□  (TM)  Technical  Manual  referral:  Instructor  corrects  the  technical  manual 

Post  Observation  BFV  Quantitative  Performance  Measures: 

For  the  Post  Observation  Analysis  section,  tally  the  total  time,  number  of  errors.  Instances  of  help-seeking  behavior,  and  the  Instances  and  types  of 
Instructor  guidance. 

Time  for  Technical  Manual  Review/Translation  into  Action: 

With  the  pre-selected  steps  Identified,  record  the  start  time  (when  the  helper  reads  the  technical  manual  step),  end  time  1  (when  the  maintalner 
begins  Implementation)  and  end  time  2  (when  the  maintalner  ends  Implementation). 


A-2 


Evaluation  Phase  1:  Set  Up 

Start  time: 


Complete 

Error  /  Backtrack 

Step 

Help-seeking  Behavior 

1.  Puts  Traverse  Drive  select  lever  into  manual  position 

□ 

2.  Puts  Gun  Elevation  Drive  into  power  and  positions  at  0  mils  or  higher 

□ 

3.  Puts  TOW  Elevation  Drive  into  manual  position 

4.  Manually  traverses  Turret  to  4750  mils 

5,  Raises  Cargo  Hatch 

□ 

6.  Zeroes  Turret  Drive  Level  on  TAS  Mounting  Plate  to  center  on  0°  (clean  mounting 
plate,  loosen  thumbscrew,  place  turret  drive  level  on  TAS  mounting  plate,  rotate  knob  on 
turret  drive  level  until  0°  level  bubble  centers,  and  tighten  thumbscrew). 

•  Start  time  (Helper  reads  step): 

•  End  time  1  (Maintainer  starts  implementation): 

•  End  time  2  (Maintainer  completes  full  implementation): 

7.  Manually  raises  TOW  launcher  to  90°  firing  position  (uses  speeder  wrench  or 
alternative  to  move  TOW  Elevation  Drive  out  of  saddle) 

□ 

8.  Zeroes  Turret  Drive  Level  UNDER  TOW  launcher  (maintainer  places  Turret  Drive 
Level  on  non-magnetic  baseplate  UNDER  TOW  launcher,  manually  raises  or  lowers 

TOW  launcher  as  helper  watches  for  90°  bubble  to  center) 
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Complete 

Error  /  Backtrack 

Step 

Help-seeking  Behavior 

9.  Zeroes  Turret  Drive  Level  on  SIDE  of  TOW  launcher  to  ensure  0°  bubble  centers 

(loosen  thumbscrew,  position  turret  drive  level  on  side  of  TOW  launcher,  rotate  knob  on 
turret  drive  level  until  1.2°  level  bubble  centers,  and  tighten  thumbscrew). 

•  Start  time  (Helper  reads  step): 

•  End  time  1  (Maintainer  starts  implementation): 

•  End  time  2  (Maintainer  completes  full  implementation): 

□ 

(Safety  precaution)  Props  up  TOW  launcher  with  jack  stand 

10.  Removes  shield  from  housing  (including  4  screws  with  socket  adapter) 

11.  Removes  plug  2W306P2  from  jack  2A204J2  to  permit  gearbox  cover  removal 

□ 

12.  Removes  gearbox  cover  including  4  screws,  lock  washers,  washers,  and  cover 
(gasket  not  removed  in  training  condition) 

End  time: 
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Post  Observation  Analysis  (BFV  Quantitative  Performance  Measures): 


Error  Tally: _ 

Help-seeking  Tally: _ 

Instructor  Guidance  Tally  (all  types): _ ;  Procedural _ ;  Conceptual: _ ;  Self-regulating: _ ;  Safety  precaution: _ ;  Technical 

manual: _ 

Total  Time:  mins. 


Conceptual  Checks: 


Question 

Answer 

Correct/Incorrect 

Steps  2-3.  "Can  you  have  the  GED  and  TED  both  in  power  or  both  in 
manual  simultaneously?  Why  or  why  not?" 

Answer:  Yes  you  may  have  both  in  power.  No  you  may  not  have  both  in 
manual  because  the  shaft  that  lifts  the  launcher  will  break. 

Step  6.  Why  is  it  important  to  ensure  that  the  initial  zeroing  of  the 
bubble  level  on  the  TAS  mounting  plate  is  exact?  “ 

Answer:  If  the  bubble  level  is  not  properly  zeroed,  the  entire  adjustment 
process  is  at  risk  of  being  ineffective  (because  the  TAS  mounting  plate  will  not 
be  appropriately  leveled  to  the  rest  of  the  vehicle). 

Step  8.  "Why  do  you  put  the  bubble  level  on  the  front  of  the  launcher 
and  not  on  the  base  plate?” 

Answer:  Because  the  base  plate  is  not  magnetic. 

Evaluation  Phase  2:  Continuity  Testing 


Start  time: 


Complete 

Error  /  Backtrack 

Step 

Help-seeking  Behavior 

13.  Installs  adapter  CA1430  on  connector  P2  of  cable  CX308 

□ 

14.  Installs  adapter  CA1430  on  jack  2A204J2  at  base  of  gearbox . 

□ 

15.  Backs  off  adjustment  screw  on  plate  until  power  is  lost  to  TOW  launcher  upper  position 
switch 

□ 

16.  Connects  multimeter  probes  to  pin  holes  B  &  C  on  the  PI  connector  of  Cable 

CX308  and  connects  multi-meter  leads  to  the  probes 
•  Start  time  (Helper  reads  step): 
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Complete 

Error  /  Backtrack 

Step 

Help-seeking  Behavior 

•  End  time  1  (Maintainer  starts  implementation): 

•  End  time  2  (Maintainer  completes  full  implementation): 
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□ 

17-18.  Manually  lowers  TOW  launcher  halfway  to  stowed  position  using  release  knob, 
and  then  raises  TOW  launcher  until  helper  says  0°  bubble  on  Turret  Drive  Level  is 

centered  on  SIDE  of  TOW  launcher 

19.  Tighten  adjustment  screw  on  plate  until  electrical  continuity  is  signaled  by 
continuous  sound  from  multi-meter 

•  Start  ti  me  (Hel  per  reads  step) : 

•  End  time  1  (Maintainer  starts  implementation): 

•  End  time  2  (Maintainer  completes  full  implementation): 

Complete 

Error  /  Backtrack 

Step 

Help-seeking 

Behavior 

20-23.  Manually  lowers  TOW  launcher  halfway  to  stowed  position  and  then  manually  raise 
TOW  launcher,  having  helper  make  sure  0°  bubble  on  Turret  Drive  Level  is  centered  again. 
Verify  continuity  exists.  If  not,  repeat  steps  18-21. 

•  Start  time  (Helper  reads  step): 

•  End  time  1  (Maintainer  starts  implementation): 

•  End  time  2  (Maintainer  completes  full  implementation): 

[Optional]  If  continuity  not  signaled,  maintainer  engages  in  troubleshooting:  check  leads  on  multi¬ 
meter,  check  cable,  repeat  raising/lowering  of  TOW  launcher,  tightening  of  adjustment  screws 

End  time: 
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Post  Observation  Analysis  (BFV  Quantitative  Performance  Measures): 

Error  Tally: _ 

Help-seeking  Tally: _ 

Instructor  Guidance  Tally  (all  types): _ ;  Procedural _ ;  Conceptual: _ ;  Self-regulating: _ ;  Safety  precaution: _ ;  Technical 

manual: _ 

Total  Time:  mins. 


Continuity  Conceptual  Checks: 


Question 

Answer 

Correct/Incorrect 

Step  15.  "Please  explain  what  happens  to  the  upper  limit 
switch  when  you  loosen  the  adjustment  screw." 

Answer:  It  breaks  the  contact  between  two  metal  plates,  interrupting  the  electrical  current  to 
the  upper  limit  switch.  It  is  OK  to  say  that  it  takes  the  ground  away  too. 

Step  17.  "Why  do  you  lower  the  TOW  launcher  halfway  to 
stowed  position  and  then  raise  it  back  up?" 

Answer:  It  is  lowered  because  this  is  when  the  upper  limit  switch  should  be  "off."  /  To  break 
continuity.  It  is  then  raised  because  this  is  the  angle  at  which  the  launcher  switch's  continuity 
should  be  restored  to  permit  the  indicator  light  to  go  on. 

Steps  20-21.  "Why  do  you  repeat  the  TOW  lifting  and 
leveling  process?" 

Answer:  To  ensure  the  adjustment  remains  fixed  even  with  movement  of  the  TOW  launcher/ 
To  double-check  continuity. 
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Evaluation  Phase  3:  Reassembly 

Start  time: 


Complete 

Error  / 
Backtrack 

Step 

Help-seeking  Behavior 

24.  Once  continuity  is  re-established,  disconnects  multimeter  probes  from 
pin  holes  B&  C  on  connector  PI  of  cable  CX308. 

25.  Removes  adapter  CA1430  from  Jack  2A204J2. 

•  Start  time  (Helper  reads  step): 

•  End  time  1  (Maintainer  starts  implementation): 

•  End  time  2  (Maintainer  completes  full  implementation): 

26.  Removes  adapter  CA1430  from  connector  P2  of  cable  CX308 

27.  Installs  cover  on  gearbox  by  installing  gasket,  cover,  4  screws  and 
washers  (no  grease  in  training  condition). 

•  Start  time  (Helper  reads  step): 

•  End  time  1  (Maintainer  starts  implementation): 

•  End  time  2  (Maintainer  completes  full  implementation): 

28.  Install  Plug  2W306P2  on  Jack  2A204J2. 

29.  Install  shield  on  housing  with  four  screws.  Torque  to  HALF  of  54-59  ft-lb 
using  torque  wrench  (training  condition). 

30.  Close  cargo  hatch. 
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Complete 

Error  / 
Backtrack 

Step 

Help-seeking  Behavior 

31.  Manually  lower  TOW  launcher  to  stowed  position  using  release  knob. 

32.  Manually  traverse  turret  back  to  6400  mils 

33.  Verify  no  faults  (that  TOW  lift  upper  position  indicator  light  is  on  when  lifting 
TOW  launcher  back  into  90°  firing  position). 

End  time: 
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Reassembly  Post  Observation  Analysis  (BFV  Quantitative  Performance  Measures): 

Error  Tally: _ 

Help-seeking  Tally: _ 

Instructor  Guidance  Tally  (all  types): _ ;  Procedural _ ;  Conceptual: _ ;  Self-regulating: _ ;  Safety  precaution: 

manual: _ 

Total  Time: _ mins. 

Reassembly  Conceptual  Checks: 


Question 

Answer 

Correct/Incorrect 

Step  33.  "Why  is  it  important  to  verify  no 
faults?" 

Answer:  To  ensure  that  the 
indicator  light  works 

All  steps.  “In  what  circumstances  will  the 
upper  limit  switch  adjustment  be 
required?” 

Answer:  When  the  upper  limit  switch  is 
malfunctioning  (or  the  tow  lift  indicator  light 
shows  an  error). 

;  Technical 


A-ll 


Post-observation  AR  Mentor  technical  questions 

We  will  use  this  to  cross-check  students’  reports  in  the  attitude  and  usability  survey. 


AR  Mentor  Condition 

Please  note  below  any  observed  difficulties  with  AR  Mentor 

1.  AR  Mentor  speech  recognition 

a.  How  often  did  the  AR  Mentor  misunderstand  the  Soldier’s 

speech? 

i.  If  the  AR  Mentor  misunderstood  the  Soldier’s 

speech,  what  was  the  nature  of  the 

misunderstanding? 

2.  Accuracy  of  the  AR  Mentor’s  instructions 

a.  Were  there  any  points  at  which  the  AR  Mentor’s 

programmed  instructions  appeared  to  be  incorrect  or 

inaccurate? 

3.  Soldier  errors 

a.  Did  the  Soldier  exhibit  signs  that  they  did  not  know  what  step 

was  next? 

i.  If  yes,  how  did  they  identify  the  next  step? 

b.  Did  the  Soldier  appear  to  be  able  to  locate  where  they 

should  be  standing  on  the  Bradley? 

c.  Did  the  Soldier  appear  to  be  looking  at  the  correct  part  on 

the  Bradley? 

d.  Did  the  solider  appear  to  understand  the  images  and 

directions  on  the  screen? 
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4.  Soldier  frustration 

a.  Did  the  Soldier  exhibit  signs  of  frustration,  boredom,  or 
discontent? 

b.  Did  the  Soldier  exhibit  signs  of  physical  discomfort  (e.g., 
problems  with  the  vest,  maintaining  3  points  of  contact)? 

c.  Did  the  Soldier  try  to  interrupt  the  AR  Mentor  at  any  point? 

5.  Helper  tasks 

a.  Did  the  helper  appear  to  know  where  to  stand  and  what 
tasks  to  do? 

i.  If  not,  when  did  these  instances  occur? 


Appendix  B:  Learning  Experience  Questionnaire 


AR  Mentor  Learning  Experience  Questionnaire 

The  questions  below  ask  you  about  your  training  experience  learning  to  adjust  the  TOW  launcher 
upper  limit  switch  of  the  Bradley  Fighting  Vehicle. 

Please  select  the  response  that  appropriately  finishes  the  following  statements  about  learning  to 
adjust  the  upper  limit  switch  of  the  Bradley  Fighting  Vehicle: 

1 .  How  mentally  demanding  was  learning  the  adjustment? 


Very  low 

1  2  3  4  5  Very  high 

2. 

How  physically  demanding  was  learning  the  adjustment? 

Very  low 

1  2  3  4  5  Very  high 

3. 

How  was  the  pace  of  learning  the  adjustment? 

Very  slow 

1  2  3  4  5  Very  fast 

4. 

How  successful  were  you  in  doing  what  you  were  asked  to  do? 

Failure 

1  2  3  4  5  Perfect 

5.  How  hard  did  you  have  to  work  to  accomplish  your  level  of 

performance? 

Very  low 

1  2  3  4  5  Very  high 

6. 

How  insecure,  discouraged,  irritated,  stressed,  and  annoyed  were 

Very  low 

1  2  3  4  5  Very  high 

7.  How  difficult  was  the  process  of  asking  questions  while  learning  the 

adjustment? 

Very  difficult 

1  2  3  4  5  Very  easy 
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8.  Please  briefly  deseribe  what  did  or  did  not  work  well  for  you  while  learning  the  adjustment 
proeedure. 


9.  Imagine  you  are  teaehing  a  fellow  maintainer  to  adjust  the  TOW  launeher  upper  limit  switeh. 
Think  about  what  you  have  learned  while  eondueting  the  adjustment.  Below  you  will  see 
pairs  of  statements  refleeting  the  knowledge  or  skills  you  need  to  do  the  adjustment.  Please 
review  eaeh  pair  and  seleet  the  ONE  statement  that  you  believe  will  represent  the  knowledge 
or  skill  that  will  be  more  diffieult  for  the  maintainer  to  learn. 


Knowledge  or  skills  needed  to  do  the  adjustment 

More 

Difficult 

a. 

making  sure  that  you  have  the  eorreet  eable  adapters  for 

the  multimeter  setup  (TC5) 

□ 

OR 

making  sure  that  the  multimeter  probes  are  eorreetly  and  eontinuously 

□ 

eonnected  through  testing  and  verifieation  ( 

b, 

manually  raising  and  lowering  the  TOW  launeher  to  adjust  it  for 

□ 

leveling  with  the  Turret  Drive  level  (TCl/3) 

OR 

elevating  the  gun  (TS4) 

□ 

e. 

knowing  whieh  steps  involved  verifying  the  problem  (TS5) 

□ 

OR 

□ 

knowing  whieh  steps  involved  ensuring  equipment  eonditions  were  met 

d, 

positioning  the  Turret  Drive  Level  on  TOP  of  the  TAS  mounting 

□ 

plate(TS4/5) 

OR 

positioning  the  Turret  Drive  Level  on  SIDE  of  TOW  launeher 

□ 

e. 

elevating  the  gun  (TS4) 

OR 

□ 

manually  traversing  the  turretTS2/l) 

□ 
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Knowledge  or  skills  needed  to  do  the  adjustment 

More 

Difficult 

f. 

manually  raising  and  lowering  the  TOW  launcher  to  adjust  it  for  leveling 

□ 

with  the  Turret  Drive  Level  (TCl/3) 

OR 

removing  and  replacing  the  metal  shield  (TSl/4) 

□ 

g- 

knowing  which  steps  involved  finding  the  part  that  caused  the  problem 

(TS5)(TS5) 

□ 

OR 

□ 

knowing  which  steps  involved  replacing  or  adjusting  the  part  (TS5)(TS5) 

h. 

making  sure  that  the  multi-meter  probes  are  correctly  and  continuously 

□ 

connected  through  testing  and  verification  (TC5) 

OR 

positioning  the  Turret  Drive  Level  on  SIDE  of  TOW  launcher  (TS4) 

□ 

i. 

knowing  which  steps  involved  checking  to  ensure  the  problem  no  longer 

exists  (TS5)(TS5) 

□ 

OR 

□ 

knowing  which  steps  involved  replacing  or  adjusting  the  part  (TS5)(TS5) 

j- 

knowing  the  circumstances  in  which  this  adjustment  will  be  required 

□ 

(P4/5) 

OR 

□ 

understanding  purpose  and  use  of  each  tool  (P4/5) 

k. 

knowing  the  risks  of  imprecise  technical  settings  (P4/5) 

□ 

OR 

understanding  the  risks  of  having  the  TOW  Elevation  Drive  (TED)  and 

Gun  Elevation  Drive  (GED)  simultaneously  in  power  position  (P3/4) 

□ 

1. 

knowing  why  you  repeat  the  TOW  lifting  and  leveling  process  (Pl/4) 

□ 

OR 

knowing  why  is  it  important  to  verify  no  faults  (Pl/5) 

□ 

m 

knowing  why  the  Turret  Drive  Eevel  is  placed  on  the  short  side  of  the 

□ 

launcher  rather  than  the  long  side  when  setting  level  (P2/3) 

OR 

knowing  why  you  repeat  the  TOW  lifting  and  leveling  process  (Pl/4) 

□ 

n. 

understanding  the  risks  of  having  the  TOW  Elevation  Drive  (TED)  and 
Gun  Elevation  Drive  (GED)  simultaneously  in  power  position  (CP3/4) 

□ 

OR 

knowing  why  is  it  important  to  verify  no  faults  Pl/5) 

□ 
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Knowledge  or  skills  needed  to  do  the  adjustment 

More 

Difficult 

0, 

understanding  the  electrical  circuitry  of  the  upper  limit  switch  of 

□ 

the  Bradley  Fighting  Vehicle  (Cl/5) 

OR 

understanding  how  loosening  the  adjustment  screw  breaks  the 

□ 

continuity  of  the  upper  limit  switch  (C2/1) 

P' 

knowing  why  you  lower  the  TOW  launcher  halfway  to  stowed  position  and 

raise  it  back  up  (C2/4) 

□ 

OR 

understanding  the  electrical  circuitry  of  the  upper  limit  switch  of 

the  Bradley  Fighting  Vehicle  Cl/5) 

□ 

q' 

understanding  how  loosening  the  adjustment  screw  breaks  the 

continuity  of  the  upper  limit  switch  (C2/1) 

□ 

OR 

knowing  why  you  lower  the  TOW  launcher  halfway  to  stowed  position  and 

raise  it  back  up  C2/4) 

□ 

Thank  you!  This  is  the  end  of  our  first  questionnaire.  Please  let  the  interviewer  know  that 
you  have  completed  it.  Please  wait  a  moment  while  they  review  the  questionnaire,  and 
follow-up  with  any  questions.  Afterwards,  they  will  give  you  the  next  questionnaire  to 
complete.  Your  feedback  will  be  very  helpful  going  forward. 


For  interviewer:  At  this  point  ask  any  follow-up  questions  based  on  your  review  of  the  survey. 
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Appendix  C:  Usability  Questionnaire 

AR  Mentor  Usability  Questionnaire 


Please  rate  your  experience  using  the  AR  Mentor  to  learn  the  adjustment  procedure  on  a  scale  of  1  to  5. 


1 .  How  easy  was  it  to  find  where  you  should  be  standing  on  the  Bradley? 


Never  could  find _ 1 _ 2 _ 3 _ 4 _ 5 _ Always  could  find 


2.  How  well  could  you  understand  your  location  from  the  Bradley  map  image? 


Poorly _ 1 _ 2 _ 3 _ 4 _ 5 _ Perfectly 


3.  How  well  could  you  follow  the  directional  arrows? 


Poorly  1  2  3  4  5  Perfectly 


4.  How  well  could  you  understand  what  you  needed  to  do  to  adjust  the  Bradley? 


Not  at  all  1  2  3  4  5  Perfectly 


5.  How  well  could  you  understand  what  to  do  from  the  3D  animations? 


Poorly  1  2  3  4  5  Perfectly 


6.  How  well  could  you  understand  what  to  do  from  the  video  demos? 


Poorly  1  2  3  4  5  Perfectly 


7.  How  well  could  you  understand  what  to  do  from  the  diagrams? 


Poorly  1  2  3  4  5  Perfectly 


8.  How  easily  could  you  read  the  AR  Mentor’s  text? 


Couldn’t  read  any _ 1 _ 2 _ 3 _ 4 _ 5 _ Could  read  all 


9.  How  well  did  you  understand  the  AR  Mentor’s  audio  instructions? 


Never  understood _ 1 _ 2 _ 3 _ 4 _ 5  Always  understood 
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10. 


How  often  did  the  AR  Mentor  know  when  you  were  speaking? 


Never _ 1 _ 2 _ 3 _ 4 _ 5  Always 


1 1 .  How  often  did  the  AR  Mentor  respond  correctly  to  what  you  were  saying? 

Never  1  2  3  4  5  Always 


12.  How  often  did  you  want  to  interrupt  the  AR  Mentor  when  it  was  saying  it  didn’t 

understand  you? 

Never  1  2  3  4  5  Always 


13.  How  often  did  you  want  to  speed  up  the  AR  Mentor  voice  when  it  was 

speaking? 


Never 


2  3  4  5  Always 


14.  How  often  did  you  want  to  slow  down  the  AR  Mentor  voice  when  it  was 

speaking? 


Never 


2  3  4  5  Always 


15.  Did  you  ask  the  AR  Mentor  to  go  to  a  specific  step?  Circle  one:  Yes  No 

15a.  If  yes,  how  well  could  the  AR  Mentor  go  to  the  step  you  wanted? 


Poorly  1  2  3  4  5  Perfectly 

16.  Did  you  ask  the  AR  Mentor  any  questions?  Circle  one:  Yes  No 

16a.  If  yes,  how  relevant  and  useful  were  the  AR  Mentor’s  answers? 

Not  at  all  1  2  3  4  5  Completely 


17.  To  what  extent  did  the  AR  Mentor  restrict  your  movement  and  get  in  the  way  of 

your  body  and  your  head? 


Always  restricted  1  2  3  4  5  Always  moved  easily 


18.  How  consistently  did  you  know  which  steps  required  you  to  coordinate  with  a  helper  and 
which  required  action  only  by  you? 


Never  knew _ 1 _ 2 _ 3 _ 4 _ 5 _ Always  knew 


19.  How  consistently  did  the  AR  Mentor  audio  and  visual  instructions  occur  when 

you  needed  them  to  occur? 


Never  1  2  3  4  5  Always 
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Thank  you!  This  is  the  end  of  our  second  questionnaire.  Please  let  the  interviewer  know  that  you  have 
completed  it,  and  they  will  be  asking  you  a  few  follow-up  questions.  Your  feedback  will  be  very  helpful  going 
forward. 
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Appendix  D:  AR  Mentor  Usability  Interview  Protocol 

AR  Mentor  Usability  Interview 

Thank  you  for  completing  the  Usability  questionnaire.  Now  I’m  going  to  ask  you  just  a  few  questions  about  using 
AR  Mentor  during  the  adjustment. 

1 .  What  type  of  media  in  AR  Mentor  did  you  use  the  most:  Audio  or  visual  (animations, 
diagrams,  video  demos,  text)?  Please  discuss  the  reasons  for  your  choice. 


2.  Please  describe  your  sense  of  the  comfort  or  discomfort  while  wearing  the  AR 

Mentor. 


3.  Please  listen  to  the  list  of  problems  and  note  whether  you  experienced  any  of 

these  and,  if  so,  please  describe  how  much  of  a  problem  they  presented  while  learning 
about  the  adjustment: 

□  Being  delayed 

□  Missing  a  step 

□  Doing  the  wrong  step 

□  Not  knowing  next  steps 

□  Being  confused  about  task 

□  Experiencing  nausea 

□  Experiencing  split  vision 

□  Feeling  cautious  and  tentative  while  moving  around  the  Bradley 

□  Catching  your  clothes,  arms,  legs  on  cables  or  cords 

□  Having  difficulty  getting  in  and  out  of  the  turret 

□  Other  (Please  describe) 


Wrap  up: 

“That  is  it  for  today.  Thank  you  for  taking  the  time  to  use  the  AR  Mentor,  and  answer  our  questions.  Your 
feedback  will  help  our  team  to  improve  the  design  and  usability  of  the  AR  Mentor.” 
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Appendix  E:  Full  Distribution  of  Data  Across  Conditions  For  Errors,  Help  Seeking,  Instructor  Guidance  and 

Total  Time 


Instructor  Guidance 

MID 

Condition 

Total 

Errors 

Total 

Help 

Seeking 

Procedural 

Conceptual 

Self 

Regulating 

Safety 

Tech 

Manual 

Total 

Guidance 

Total 

Time 

Pl.l 

AR 

Mentor 

5 

15 

4 

0 

0 

0 

1 

6 

1:52:00 

P1.2 

AR 

Mentor 

2 

3 

1 

0 

0 

0 

0 

1 

1:04:00 

P2.2 

AR 

Mentor 

3 

11 

1 

0 

0 

0 

0 

1 

1:02:00 

P2.3 

AR 

Mentor 

1 

1 

0 

0 

0 

0 

0 

0 

0:46:00 

Mean 

2.75 

7.50 

1.50 

0.00 

0.00 

0.00 

0.25 

2.00 

1:11:00 

Total 

11.00 

30.00 

6.00 

0.00 

0.00 

0.00 

1.00 

8.00 

4:44:00 

P3.2 

Instructor 

1 

9 

37 

1 

17 

5 

1 

58 

0:55:00 

P4.1 

Instructor 

2 

7 

23 

0 

9 

1 

0 

35 

0:56:00 

Mean 

1.5 

8 

30 

0.5 

13 

3 

0.5 

46.5 

0:55:30 

Total 

3.00 

16.00 

60.00 

1.00 

26.00 

6.00 

1.00 

93.00 

1:51:00 

P5.1 

Tech 

Manual 

11 

25 

15 

0 

6 

5 

2 

26 

2:30:00 

P6.2 

Tech 

Manual 

5 

25 

14 

0 

1 

2 

0 

19 

2:00:00 

Mean 

8.00 

25.00 

14.50 

0.00 

3.50 

3.50 

1.00 

22.50 

2:15:00 

Total 

16.00 

50.00 

29.00 

0.00 

7.00 

7.00 

2.00 

45.00 

4:30:00 
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Appendix  F,  Instructor  Interview  Protocol 


AR  Mentor  Practice  and  Interview  Protocol  -  Bradley  Instrnctor 


Instrnctions: 


Our  team  would  like  to  draw  upon  your  expertise  teaching  mechanics  to  adjust  the  Bradley  to  help  us  improve  on 
the  design  and  usability  of  new  training  technology  called  Augmented  Reality  Mentor  or  AR  Mentor.  I  will  be 
asking  you  to  conduct  an  adjustment  on  the  Bradley  so  you  can  have  the  experience  that  a  trainee  would  with  the  AR 
Mentor,  and  then  I  will  interview  you.  The  adjustment  may  take  between  40-50  minutes  to  complete.  The  interview 
will  take  about  20-25  minutes. 

The  SRI  team  will  take  5  minutes  to  train  you  in  how  to  use  the  AR  Mentor.  [Use  the  training  script  to  familiarize 
the  instructor  with  the  features  of  the  AR  Mentor.] 

You  will  be  conducting  the  upper  limit  switch  TOW  launcher  adjustment  on  the  Bradley  with  the  assistance  of  the 
AR  Mentor.  Is  this  an  adjustment  you  have  taught?  (Circle  answer  below) 

Yes  No 

As  you  go  through  the  adjustment,  the  SRI  team  will  assist  you  should  you  have  problems  with  this  new  technology. 
I  will  observe  you  as  you  conduct  the  adjustment  and  make  notes  of  your  reactions  to  the  technology.  {During  the 
adjustment,  please  observe  and  note  reactions  and  questions) 

Instructor  has  now  completed  the  adjustment: 

Now  that  you  have  completed  the  adjustment.  I’d  like  to  ask  you  some  questions  about  your  experience.  I  will  be 
taking  notes  and  audio  recording  our  interview.  Please  answer  as  honestly  as  possible,  as  your  feedback  is  very 
important. 

Questions: 

Referring  to  your  observation  notes,  ask  the  instructor  about  any  salient  reactions,  positive  and/or  negative  (e.g., 
frequent  positive  comments,  frequent  interaction  problems,  etc.) 

1.  As  I  observed  you  working,  it  appeared  that  you  liked/disliked  [name  the  AR  Mentor  features  that  were  subjects 
of  positive/negative  reactions].  Is  this  correct?  As  an  instructor,  what  struck  you  about  these  features?  How  do 
you  think  your  students  would  react  to  these  features?  [go  through  each  observed  feature  listed  one  at  a  time] 
Follow  up:  What  was  your  impression  of  how  the  adjustment  process  flowed  using  the  AR  Mentor? 

2.  Please  describe  your  sense  of  comfort  or  discomfort  while  wearing  the  AR  Mentor.  Do  you  think  your 
students’  comfort  level  would  be  adequate  with  the  AR  Mentor? 

3.  Based  on  your  experience,  how  might  your  work  as  an  instructor  change  if  your  students  had  access  to  the  AR- 
Mentor?  (Follow  up:  Do  you  think  students  will  remember  the  procedure  and  steps  better  using  the  AR  Mentor 
than  the  TM?) 

4.  Were  there  any  features  of  the  AR  Mentor  that  you  thought  would  benefit  your  students?  {Follow-up  prompts: 
ask  about  3D  animations,  videos,  diagrams,  text,  voice-overs) 

5.  Were  there  any  features  of  the  AR-Mentor  that  you  thought  your  students  might  find  inconvenient  or  that  might 
require  special  training?  {Follow  up  prompts:  Ask  about  eye-gaze  tracking  arrow,  3D  animations,  videos, 
diagrams,  text,  voice-overs) 
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6.  Was  there  any  way  that  the  instructions  in  the  AR  Mentor  could  have  been  improved?  Was  there  any 
information  or  instruction  lacking  that  could  confuse  students?  (Follow-up:  Probe  for  should  information  be 
focused  on  new  skills  to  this  task,  such  as  Turret  Drive  Level,  and  should  information  on  older  skills  be  de- 
emphasized,  such  as  multimeter  connections,  turret  rotation?) 

7.  Do  you  think  that  all  the  AR  Mentor  instructional  visualizations  (diagrams,  3D  animations,  videos)  were 
appropriate  tools  for  teaching  the  tasks?  (Follow-up:  if  the  instructor  didn ’t  think  all  were  appropriate,  probe 
for  which  ones  were  not  appropriate  and  why.  Probe  for:  Is  the  visualization  necessary  to  support  learning  or 
not?  Are  there  some  places  where  visualization  is  needed  and  not  present?) 

8.  As  you  used  the  AR  Mentor,  did  you  have  some  ideas  about  ways  you  might  be  able  to  use  the  AR  Mentor  or  its 
visualizations  and  instructions  in  your  own  teaching  or  demonstrations?  {Follow  up  questions:  Probe  for  how 
to  support  the  students  using  the  AR  Mentor,  how  to  correct  student  performance,  how  to  grade  student 
performance,  and  how  to  explain  aspects  of  the  adjustment) 

9.  Did  you  learn  anything  from  using  the  AR  Mentor  that  you  would  incorporate  into  your  own  teaching  of  the 
adjustment? 


Wrap-up: 

That  is  the  end  of  our  interview,  thank  you  for  taking  the  time  to  use  the  AR  Mentor,  and  answer  our  questions. 
Your  feedback  will  be  very  helpful  going  forward. 
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Appendix  G:  Mechanic  Interview  Protocol 

AR  Mentor  Practice  and  Interview  Protocol  -  Experienced  Mechanic 


Instrnctions: 


Our  team  would  like  to  draw  upon  your  expertise  as  a  mechanic  to  help  us  improve  on  the  design  and  usability  of 
new  training  technology  called  Augmented  Reality  Mentor  or  AR  Mentor.  I  will  be  asking  you  to  conduct  an 
adjustment  on  the  Bradley  so  you  can  have  the  experience  that  an  experienced  mechanic  would  have  learning  a  new 
adjustment  with  the  AR  Mentor,  and  then  I  will  interview  you.  The  adjustment  may  take  between  40-50  minutes  to 
complete.  The  interview  will  take  about  20-25  minutes. 

The  SRI  team  will  take  5  minutes  to  train  you  in  how  to  use  the  AR  Mentor.  [Use  the  training  script  to  familiarize 
the  instructor  with  the  features  of  the  AR  Mentor.] 

You  will  be  conducting  the  upper  limit  switch  TOW  launcher  adjustment  on  the  Bradley  with  the  assistance  of  the 
AR  Mentor.  Is  this  an  adjustment  you  have  ever  done?  (Circle  answer  below) 

Yes  No 

As  you  go  through  the  adjustment,  the  SRI  team  will  assist  you  should  you  have  problems  with  this  new  technology. 
I  will  observe  you  as  you  conduct  the  adjustment  and  make  notes  of  your  reactions  to  the  technology.  {During  the 
adjustment,  please  observe  and  note  reactions  and  questions). 

Mechanic  has  now  completed  the  adjustment: 

Now  that  you  have  completed  the  adjustment.  I’d  like  to  ask  you  some  questions  about  your  experience.  I  will  be 
taking  notes  and  audio  recording  our  interview.  Please  answer  as  honestly  as  possible,  as  your  feedback  is  very 
important. 


Questions: 

Referring  to  your  observation  notes,  ask  the  mechanic  about  any  salient  reactions,  positive  and/or  negative  (e.g., 

frequent  positive  comments,  frequent  interaction  problems,  etc.) 

1.  As  I  observed  you  working,  it  appeared  that  you  liked/disliked  [name  the  AR  Mentor  features  that  were  subjects 
of  positive/negative  reactions].  Is  this  correct? 

a.  As  an  experienced  mechanic,  what  struck  you  about  these  features?  How  do  you  think  other  experienced 
mechanics  would  react  to  these  features  as  they  learn  a  new  adjustment  in  the  field?  [Go  through  each 
observed  feature  listed  one  at  a  time] 

b.  Follow  up:  What  was  your  impression  of  how  the  adjustment  process  flowed  using  the  AR  Mentor? 

2.  Please  describe  your  sense  of  comfort  or  discomfort  while  wearing  the  AR  Mentor.  Do  you  think  other 
mechanics  would  find  the  AR  Mentor  comfortable  to  use  in  the  field? 

3.  As  an  experienced  mechanic,  describe  the  prior  knowledge  you  used  to  complete  the  task  in  this  unfamiliar 
situation. 

4.  Did  the  AR  Mentor  help  you  overcome  your  unfamiliarity  with  the  Bradley  and  ultimately  complete  the 
adjustment?  Would  some  preparation  or  overall  orientation  to  the  Bradley  have  been  helpful  in  facilitating  the 
completion  of  the  adjustment?  (Follow-up:  Probe  for  what  information,  preparation,  or  orientation  would  have 
been  helpful.) 
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5.  Based  on  your  past  experience  in  the  field,  do  you  think  you  might  be  able  to  use  the  AR-Mentor  to  learn  how 
to  perform  new  adjustments? 

6.  Were  there  any  features  of  the  AR  Mentor  that  would  particularly  benefit  mechanics  working  out  in  the  field? 
{Follow-up:  Probe  for  3D  animations,  videos,  diagrams,  text,  voice-overs) 

7.  Were  there  any  features  of  the  AR-Mentor  that  you  thought  mechanics  in  the  field  might  find  inconvenient  or 
that  might  require  special  training?  {Follow-up:  Probe  for  eye-gaze  tracking  arrow,  3D  animations,  videos, 
diagrams,  text,  voice-overs) 

8.  How  could  the  instructions  given  by  AR  Mentor  have  been  improved  for  an  experienced  mechanic?  Was  there 
any  information  or  instruction  lacking?  Was  there  too  much  information  and  not  enough  flexibility,  such  as 
skipping  steps  or  going  to  only  certain  steps?  (Follow-up:  Probe  for  which  skills  seemed  particularly  new  to 
the  experienced  mechanic  and  which  seemed  to  be  more  familiar,  e.g.,  multimeter  connections,  turret  rotation) 

9.  Do  you  think  that  all  the  AR  Mentor  instructional  visualizations  (diagrams,  3D  animations,  videos)  were 
appropriate  tools  for  learning  the  tasks?  (Follow-up:  If  the  mechanic  didn ’t  think  all  were  appropriate,  probe 
for  which  ones  were  not  appropriate  and  why.  Also  probe  for:  Is  the  visualization  necessary  to  support 
learning  or  not?  Are  there  some  places  where  visualization  is  needed  and  not  present?) 

10.  Do  you  think  that  doing  the  adjustment  with  the  AR  Mentor  aided  you  in  remembering  the  steps  and  the  overall 
procedure?  Do  you  feel  that  learning  was  sufficiently  effective  so  that  you  would  not  need  to  use  the  AR 
Mentor  the  next  time  you  conduct  this  adjustment? 

a.  Would  it  have  been  more  efficient  to  use  a  technical  manual  to  learn  the  new  adjustment,  instead  of  using 
the  AR  Mentor?  In  the  future  when  doing  the  adjustment,  would  you  use  the  AR  Mentor  or  the  TM? 

1 1 .  Are  there  advantages  or  disadvantages  to  using  the  AR  Mentor  to  perform  the  adjustment,  as  opposed  to  a 
technical  manual? 

12.  What  advantages,  if  any,  do  you  see  in  using  the  AR  Mentor  in  in  the  field  over  using  only  a  TM  and  guidance 
from  another  experienced  mechanic? 

13.  Would  you  recommend  using  AR  Mentor  for  repairs  or  adjustments  to  other  mechanics  in  your  field?  Please 
explain  why  or  why  not. 


Wrap-up: 

That  is  the  end  of  our  interview,  thank  you  for  taking  the  time  to  use  the  AR  Mentor,  and  answer  our  questions. 
Your  feedback  will  be  very  helpful  going  forward. 
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